{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6eac35c8-0c15-4279-b17b-3e31a9a95c57",
   "metadata": {},
   "source": [
    "# 【A/B测试 支付宝营销策略效果分析】"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ce61925-2361-4fc1-8a47-ab700a650c92",
   "metadata": {},
   "source": [
    "A/B测试常用于比较不同设计、运营方案的优劣，以辅助决策。本文以支付宝营销活动为例，通过广告点击率指标比较两组营销策略的广告投放效果。\n",
    "\n",
    "## 1. 数据来源\n",
    "\n",
    "本文所用数据集来自阿里云天池：  \n",
    "[阿里云天池 - Audience Expansion Dataset](https://tianchi.aliyun.com/dataset)\n",
    "\n",
    "该数据集包含三张表，分别记录了支付宝两组营销策略的活动情况：\n",
    "\n",
    "- `emb_tb_2.csv`：用户特征数据集\n",
    "- `effect_tb.csv`：广告点击情况数据集\n",
    "- `seed_cand_tb.csv`：用户类型数据集\n",
    "\n",
    "本分析报告主要使用广告点击情况数据，涉及字段如下：\n",
    "\n",
    "- `dmp_id`：营销策略编号（源数据文档未作说明，这里根据数据情况设定为1：对照组，2：营销策略一，3：营销策略二）\n",
    "- `user_id`：支付宝用户ID\n",
    "- `label`：用户当日在是否点击活动广告（0：未点击，1：点击）"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb0f633f-1c27-44fd-9c34-cc8a66f833eb",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 2. 数据处理\n",
    "\n",
    "### 2.1 数据导入和清洗\n",
    "\n",
    "1. 整合表\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "0ee3718a-0c17-4f48-8a28-23e694614184",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "48879b19-49e0-465e-b95c-c365634bb0b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load data\n",
    "data=pd.read_csv('effect_tb.csv',header=None)\n",
    "data.columns=['dt','user_id','label','dmp_id']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0b740dc0-9ebd-4686-94e0-ece981bb7bc1",
   "metadata": {},
   "outputs": [],
   "source": [
    "#删掉日志属性\n",
    "data=data.drop(columns='dt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "6bf647d1-a0ed-4637-bbca-cb35b960edb7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>label</th>\n",
       "      <th>dmp_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>2.645958e+06</td>\n",
       "      <td>2.645958e+06</td>\n",
       "      <td>2.645958e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>3.112995e+06</td>\n",
       "      <td>1.456297e-02</td>\n",
       "      <td>1.395761e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>1.828262e+06</td>\n",
       "      <td>1.197952e-01</td>\n",
       "      <td>6.920480e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1.526772e+06</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>3.062184e+06</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>4.721132e+06</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>2.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>6.265402e+06</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>3.000000e+00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            user_id         label        dmp_id\n",
       "count  2.645958e+06  2.645958e+06  2.645958e+06\n",
       "mean   3.112995e+06  1.456297e-02  1.395761e+00\n",
       "std    1.828262e+06  1.197952e-01  6.920480e-01\n",
       "min    1.000000e+00  0.000000e+00  1.000000e+00\n",
       "25%    1.526772e+06  0.000000e+00  1.000000e+00\n",
       "50%    3.062184e+06  0.000000e+00  1.000000e+00\n",
       "75%    4.721132e+06  0.000000e+00  2.000000e+00\n",
       "max    6.265402e+06  1.000000e+00  3.000000e+00"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head(3)\n",
    "data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "828f1b0c-18f5-43c4-be25-5ad2c2839db0",
   "metadata": {
    "tags": []
   },
   "source": [
    "2.重复值处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "13bf8f0f-30da-44fa-be70-d7507d58f715",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2645958, 3)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#shape of data\n",
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "9c903546-0fe8-46fa-abc7-7d2f8a0e9268",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "user_id    2410683\n",
       "label            2\n",
       "dmp_id           3\n",
       "dtype: int64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#distinct count of data\n",
    "data.nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15c0a374-bd47-46b3-a79a-954de348179c",
   "metadata": {},
   "source": [
    "检测是否存在重复行"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "7a72da1e-04ab-4fc8-8097-8d12a448e007",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>label</th>\n",
       "      <th>dmp_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8529</th>\n",
       "      <td>1027</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1485546</th>\n",
       "      <td>1027</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1579415</th>\n",
       "      <td>1471</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>127827</th>\n",
       "      <td>1471</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>404862</th>\n",
       "      <td>2468</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1382121</th>\n",
       "      <td>6264633</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1382245</th>\n",
       "      <td>6264940</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2575140</th>\n",
       "      <td>6264940</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1382306</th>\n",
       "      <td>6265082</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2575171</th>\n",
       "      <td>6265082</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>25966 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         user_id  label  dmp_id\n",
       "8529        1027      0       1\n",
       "1485546     1027      0       1\n",
       "1579415     1471      0       1\n",
       "127827      1471      0       1\n",
       "404862      2468      0       1\n",
       "...          ...    ...     ...\n",
       "1382121  6264633      0       1\n",
       "1382245  6264940      0       1\n",
       "2575140  6264940      0       1\n",
       "1382306  6265082      0       3\n",
       "2575171  6265082      0       3\n",
       "\n",
       "[25966 rows x 3 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[data.duplicated(keep=False)].sort_values(by=['user_id'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "b31086f3-574c-413e-a034-4295ec0e7502",
   "metadata": {},
   "outputs": [],
   "source": [
    "#删除重复行\n",
    "data=data.drop_duplicates()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "b3036427-ea56-4bd4-9de8-dd6c3a957fcc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>label</th>\n",
       "      <th>dmp_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [user_id, label, dmp_id]\n",
       "Index: []"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#检查还有没有重复行\n",
    "data[data.duplicated(keep=False)]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba068848-4fb0-4569-b741-3c2767f238c9",
   "metadata": {},
   "source": [
    "空值处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "f594b8f6-f989-4355-bb01-e15fd86a0fa5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "user_id    0\n",
       "label      0\n",
       "dmp_id     0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#检查缺失值\n",
    "data.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efb5029a-a5a5-448c-b43e-076c0c35e6e2",
   "metadata": {},
   "source": [
    "异常值检查，通过透视图表查看各属性字段是否存在不合理取值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "3a799159-df92-4fbb-a0f8-4f077969f213",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>label</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>All</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dmp_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1881745</td>\n",
       "      <td>23918</td>\n",
       "      <td>1905663</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>404811</td>\n",
       "      <td>6296</td>\n",
       "      <td>411107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>307923</td>\n",
       "      <td>8282</td>\n",
       "      <td>316205</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>All</th>\n",
       "      <td>2594479</td>\n",
       "      <td>38496</td>\n",
       "      <td>2632975</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "label         0      1      All\n",
       "dmp_id                         \n",
       "1       1881745  23918  1905663\n",
       "2        404811   6296   411107\n",
       "3        307923   8282   316205\n",
       "All     2594479  38496  2632975"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.pivot_table(index='dmp_id',columns='label',values='user_id',aggfunc=\"count\",margins='True')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56e64901-be11-460f-b3b9-0ec6bd79461a",
   "metadata": {},
   "source": [
    "dmp_id只有1、2、3，不存在异常值，不需要进行异常值处理"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31a56c20-b663-45f2-afae-056fd36282e3",
   "metadata": {},
   "source": [
    "5. 数据类型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "db50c5d2-e88b-41f2-8f8b-ccfc82dd88ee",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "user_id    int64\n",
       "label      int64\n",
       "dmp_id     int64\n",
       "dtype: object"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b1e457b-52dd-4038-93d6-3f86e80c7248",
   "metadata": {},
   "source": [
    "数据类型正常，不需要处理"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "916ba694-98b1-4bfc-83c9-90d89f3f111d",
   "metadata": {},
   "source": [
    "### 2.2 样本容量检测\n",
    "在进行 A/B 测试前，需检查样本容量是否满足试验所需最小值。\n",
    "\n",
    "这里借助 Evan Miller 的样本量计算工具：  \n",
    "[Sample Size Calculator](https://www.evanmiller.org/ab-testing/sample-size.html)\n",
    "\n",
    "首先需要设定点击率基准线以及最小提升比例，我们将对照组的点击率设为基准线。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "6ad4b970-35b0-4d2d-ba40-fe7f58887111",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.012551012429794775"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#对照组的点击率计算\n",
    "data[data['dmp_id']==1][\"label\"].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c65dc3f-3072-4ae4-9b6e-11e42849b5b2",
   "metadata": {},
   "source": [
    "对照组点击率为 1.26%，假定我们希望新的营销策略能让广告点击率至少提升 1 个百分点，则算得所需最小样本量为：2167。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "a2ebe6cc-d98c-4d97-9bf3-82b318ac2b44",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dmp_id\n",
       "1    1905663\n",
       "2     411107\n",
       "3     316205\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#营销策略的样本量\n",
    "data['dmp_id'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46da9302-1469-4c65-a963-e1918d1d0eb0",
   "metadata": {},
   "source": [
    "两组营销策略的样本量分别为41万和31万，满足最小样本量需求"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "0c314696-4bd0-4b4e-bae6-9eb5e651c10b",
   "metadata": {},
   "outputs": [],
   "source": [
    "#清洗好的数据备份\n",
    "data.to_csv('output.csv',index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "e0b0dee3-83a9-45f5-8e89-07c0e3559320",
   "metadata": {},
   "outputs": [],
   "source": [
    "data=pd.read_csv('output.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8e19de7-d991-46b0-8321-cea4fac4aa5f",
   "metadata": {},
   "source": [
    "# 假设检验\n",
    "观察几组点击率的情况"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "a8880075-d7c9-45e5-b284-b4634522e80d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "对照组： 0.012551012429794775\n",
      "营销策略1： 0.015314747742072015\n",
      "营销策略2： 0.026191869198779274\n"
     ]
    }
   ],
   "source": [
    "print(\"对照组：\",data[data[\"dmp_id\"]==1][\"label\"].mean())\n",
    "print(\"营销策略1：\",data[data[\"dmp_id\"]==2][\"label\"].mean())\n",
    "print(\"营销策略2：\",data[data[\"dmp_id\"]==3][\"label\"].mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dd58b9d-9062-4b07-8248-7032ae1c5f5a",
   "metadata": {},
   "source": [
    "可以看到策略一和策略二相较对照组在点击率上都有不同程度提升。\n",
    "\n",
    "其中策略一提升 0.2 个百分点，策略二提升 1.3 个百分点，只有策略二满足了前面我们对点击率提升最小值的要求。\n",
    "\n",
    "接下来需要进行假设检验，看看策略二点击率的提升是否显著。\n",
    "\n",
    "a. **零假设和备择假设**\n",
    "\n",
    "记对照组点击率为 p1，策略二点击率为 p2，则：\n",
    "\n",
    "- 零假设 H0： p1 ≥ p2\n",
    "- 备择假设 H1： p1 < p2\n",
    "\n",
    "b. **分布类型、检验类型和显著性水平**\n",
    "\n",
    "样本服从二项分布，独立双样本，样本大小 n > 30，总体均值和标准差未知，所以采用 Z 检验。  \n",
    "显著性水平 α 取 0.05。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c1ed66d-ac89-485f-aa06-88e8a49f2d77",
   "metadata": {},
   "source": [
    "### 3.1 方法一：公式计算\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "8f7e3d18-9a50-4e69-92f2-5f46dc23eb06",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "总和点击率:  0.014492310074225832\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_894/757505842.py:6: UserWarning: Boolean Series key will be reindexed to match DataFrame index.\n",
      "  c_old = len(data[data.dmp_id == 1][data.label == 1])\n",
      "/tmp/ipykernel_894/757505842.py:7: UserWarning: Boolean Series key will be reindexed to match DataFrame index.\n",
      "  c_new = len(data[data.dmp_id == 3][data.label == 1])\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# 用户数\n",
    "n_old = len(data[data.dmp_id == 1])  # 对照组\n",
    "n_new = len(data[data.dmp_id == 3])  # 策略二\n",
    "\n",
    "# 点击数\n",
    "c_old = len(data[data.dmp_id == 1][data.label == 1])\n",
    "c_new = len(data[data.dmp_id == 3][data.label == 1])\n",
    "\n",
    "# 计算点击率\n",
    "r_old = c_old / n_old\n",
    "r_new = c_new / n_new\n",
    "\n",
    "# 总和点击率\n",
    "r = (c_old + c_new) / (n_old + n_new)\n",
    "\n",
    "print(\"总和点击率: \", r)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "e2f416dc-9122-493c-9574-5675a2332652",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "检验统计量 z:  -59.44168632985996\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# 计算检验统计量 z\n",
    "z = (r_old - r_new) / np.sqrt(r * (1 - r) * (1/n_old + 1/n_new))\n",
    "\n",
    "print(\"检验统计量 z: \", z)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "2eb79e70-dc0d-49fb-b3e1-fa8df77fbbb9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1.6448536269514729"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 查 α = 0.05 对应的 z 分位数\n",
    "from scipy.stats import norm\n",
    "z_alpha = norm.ppf(0.05)\n",
    "z_alpha\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f5c35ed-e88f-442b-818e-15afcc2c7cc8",
   "metadata": {},
   "source": [
    "_z_alpha = -1.64_，检验统计量 _z = -59.44_，该检验为左侧单尾检验，拒绝域为 { z < z_alpha }。\n",
    "\n",
    "所以我们可以得出结论：原假设不成立，策略二点击率的提升在统计上是显著的。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75e233f6-c10e-4ed4-ac44-ffb740b747cb",
   "metadata": {},
   "source": [
    "### 3.2 方法二：python函数计算"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "4d354082-a83a-440e-b5f7-e7385c7afeda",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "检验统计量 z:  -59.44168632985996 , p 值:  0.0\n"
     ]
    }
   ],
   "source": [
    "import statsmodels.stats.proportion as sp\n",
    "\n",
    "z_score, p = sp.proportions_ztest([c_old, c_new], [n_old, n_new], alternative='smaller')\n",
    "\n",
    "print(\"检验统计量 z: \", z_score, \", p 值: \", p)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "009b234a-ecff-415d-9680-c7c4afe2ee50",
   "metadata": {},
   "source": [
    "p 值约等于 0，p < α，与方法一结论相同，拒绝原假设。\n",
    "\n",
    "作为补充，我们再检验下策略一的点击率提升是否显著。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "75555bb8-aa42-4078-b264-a5444f4fb932",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "检验统计量 z:  -14.165873564308429 , p 值:  7.450121742737582e-46\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_894/4114044008.py:3: UserWarning: Boolean Series key will be reindexed to match DataFrame index.\n",
      "  [c_old, len(data[data.dmp_id == 2][data.label == 1])],\n"
     ]
    }
   ],
   "source": [
    "# 策略一检验\n",
    "z_score, p = sp.proportions_ztest(\n",
    "    [c_old, len(data[data.dmp_id == 2][data.label == 1])],\n",
    "    [n_old, len(data[data.dmp_id == 2])],\n",
    "    alternative='smaller'\n",
    ")\n",
    "\n",
    "print(\"检验统计量 z: \", z_score, \", p 值: \", p)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f211fbb1-cb9c-492b-ab3b-1cd205be8289",
   "metadata": {},
   "source": [
    "p 值约等于 7.45e-46，p < α，但因为前面我们设置了对点击率提升的最小值要求，这里仍只选择第二组策略进行推广。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60af2321-38b8-42b0-8697-07882d6b2779",
   "metadata": {},
   "source": [
    "## 4. 结论\n",
    "\n",
    "综上所述，两种营销策略中，策略二对广告点击率有显著提升效果，且相较于对照组点击率提升了近一倍，因而在两组营销策略中应选择第二组进行推广。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6f63f9f-db24-40b4-9110-1307c0861d0b",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
